Most computer applications (hereinafter “applications”) are very complex systems that, due to their complexity, require significant testing to ensure that the application will execute as desired.
To facilitate the testing of applications, test cases or test suites (essentially a collection of test cases) are designed, implemented and used to test a portion or the whole of an application (often referred to as the subject under test). In many applications, these test cases manipulate the external facade or interface of the subject under test. The results of these test cases are then analyzed and evaluated. As many applications are quite complex, several, sometimes hundreds, of test cases are used for the testing of a single application. A “test case” is used herein to describe the type, subject under test and items to be tested. A test case is implemented (in a scripting or programming language) to create a “test script”.
For example, a database application may need to be tested to determine whether data can be added to the database (this is the test case). A test script would need to be created to implement the test case. The exemplary test script could include several steps, instructions or processes to test this aspect of the application may include: gaining access to update the database, transmitting the update request; receiving confirmation that the update request has been executed; reading from the database to determine if the data in the update request was stored successfully; and then logging off from the database.
As a result of execution, a test script execution will generate an output which indicates whether the application successfully or unsuccessfully performed the test case (the test output).
Modern testing tools and frameworks have been developed so that test cases and test scripts can be created, modified and managed. Test cases and test scripts are designed to improve the quality of the subject under test (e.g., reduce the number of bugs; eliminate or reduce the potential for catastrophic failure—“crashing”; improve failure recovery; etc.). Unfortunately, for many software applications, application quality has often been overshadowed by the desire to provide more and richer features. However, as the reliance on software applications increases and software is used in many more critical situations where lack of software application quality is simply not acceptable, reliability and quality has become far more important. For example, one methodology that is being promoted in the application development community is known as “Extreme Programming”. Extreme programming is a software development methodology that is based on values of simplicity, communication and feedback. One of the core principles recommends that the test cases and scripts be created before the actual software application (which will be tested by the test scripts) is created. This extreme programming methodology results (as does other methodologies) in the creation of large numbers of tests cases.
In the case where the software application is created using object oriented programming, there are often several test cases and scripts developed for a single object. A single software application may involve thousands or tens of thousands (or more) objects. As a result of both of these factors (improved focus on software application and new programming methodologies), the test case base (i.e., the collection of test cases and scripts for an application) becomes very large and complex. As a result of the size and complexity of the test case base, the logistics of managing the test cases and the scripts which implement these test cases becomes very onerous to the development team. For example, it may be difficult to ascertain whether: a script exists which corresponds to a test case; to which part of the application a test case or test script pertains; etc. Also, developers are often under pressure to use test cases and scripts to test components and system level testing using cases and scripts designed for other scenarios. This type of misapplication of test cases and scripts often results in portions of the application (e.g., components) being only partially tested. Further, as developers and test team members develop test cases and scripts either completely manually or using a combination of test tools which automate the generation of test scripts together with manual modifications or extensions, test scripts and cases developed by different people often have very different structures and quality. Finally, while new methodologies may be developed to address some of these shortcomings involving test cases and scripts, such methodologies would not address the shortcomings which exist in test cases and scripts which already exist.
Accordingly, addressing, at least in part, the shortcomings noted above, thus improving the testing of computer applications, is desired.